User scanning and one-way augmented reality viewing system

ABSTRACT

A method for providing augmented reality content includes identifying a customer at a customer interaction location, retrieving customer-specific interaction data, determining locations of a customer service representative in an area adjacent to a first side of a transparent panel and the customer in an area adjacent to a second side of the transparent panel, and projecting an augmented reality element to the first side of a transparent panel, the augmented reality element displaying the customer-specific interaction data on the first side of the transparent panel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.16/915,009, filed on Jun. 29, 2020, and titled “User Scanning andOne-Way Augmented Reality Viewing System,” which claims priority to U.S.Provisional Patent Application No. 62/869,715, filed on Jul. 2, 2019,and titled “User Scanning and One-Way Augmented Reality Viewing System,”the disclosure of which is hereby incorporated by reference in itsentirety.

TECHNICAL FIELD

The present disclosure relates generally to user detection andauthentication systems, including sensor systems to detect and identifyusers. The present disclosure also relates to one-way augmented realitysystems to provide information for interacting with users.

BACKGROUND

Customer service representatives at locations such as banks or otherfinancial institutions, as well as other retail locations (e.g., stores,restaurants, etc.), interact with many customers on a daily basis. Insome cases, the interactions may be live, where the customers and thecustomer service representatives are at the same location. In othercases, the customer service representatives may be at a location remotefrom the customer location. In either case, a customer servicerepresentative may have no knowledge regarding the identity or motivesof a person entering the location. For example, a customer may be aclient, e.g., a person that frequently visit a particular location, suchas a bank. Alternatively, a customer may be a prospective client seekingthe services provided at the location. In other cases, the customer maybe a fraudster attempting to defraud a business, for example, by passinga bad check mitigate risk of fraud, money laundering and other forms oftheft. In any of these scenarios, information regarding the personentering the location as well as the person's motives may not beavailable to the customer service representatives to enable them to takeappropriate action with regard to a particular customer.

SUMMARY

In one example, an augmented reality user interaction system may includea processing unit including a processor, a camera device configured tocapture sensor data about a location, and to capture visual data aboutan object in the location, a computer-readable memory having storedthereon instructions that are executable by the processor. Theinstructions may cause the system to receive the sensor datacorresponding to the location from the camera device, the sensor dataincluding the visual data, analyze the sensor data to detect a userwithin a proximity of the location, detect, based on analyzing thesensor data, a characteristic of the user, compare the characteristic ofthe user to a data store having a plurality of user profiles with usercharacteristics stored therein for a plurality of users, identify a userprofile among the plurality of user profiles by matching thecharacteristic to a user characteristic associated with the userprofile, and generate and output user interaction data associated withthe user profile via an augmented reality system.

In another example, an augmented reality customer interaction system mayinclude a transparent panel having a first side and a second side thatis opposite to the first side, and a camera device configured to capturevisual data from an area adjacent to the second side of the transparentpanel. The visual data may include identifying features of a customerlocated in the area with respect to the second side of the transparentpanel. The system may further include a projection system configured toproject information on the first side of the transparent panel. Theinformation projected on the first side of the transparent panel mayinclude customer interaction data retrieved from a data store based onthe identifying features of the customer.

In a further example, a method for providing augmented reality contentmay include identifying a customer at a customer interaction location,retrieving customer-specific interaction data, determining locations ofa customer service representative in an area adjacent to a first side ofa transparent panel and the customer in an area adjacent to a secondside of the transparent panel, and projecting an augmented realityelement to the first side of a transparent panel, the augmented realityelement displaying the customer-specific interaction data on the firstside of the transparent panel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a customer serviceenvironment according to some aspects of the present disclosure;

FIG. 2 is a diagram illustrating another example of a similar oridentical customer service environment according to some aspects of thepresent disclosure;

FIG. 3 is a diagram illustrating an example of a distributed computingenvironment according to some aspects of the present disclosure;

FIG. 4 is a block diagram illustrating an example embodiment of acomputer server according to some aspects of the present disclosure;

FIG. 5 is a block diagram of an illustrative computer system accordingto some aspects of the present disclosure;

FIG. 6 is a flowchart illustrating an example of a process foridentifying a user at a location based on multiple sensors andmulti-factor user characteristic matching according to some aspects ofthe present disclosure;

FIGS. 7A and 7B illustrate examples of collecting visual facial and earrecognition data according to aspects of the present disclosure;

FIG. 8 is a flowchart illustrating an example of a process forgenerating and providing augmented reality content (and/or mixed realitycontent) to one side of a transparent panel according to aspects of thepresent disclosure; and

FIG. 9 is a diagram illustrating an example of an augmented realitycustomer user interface that may be provided to a customer servicerepresentative according to some aspects of the present disclosure.

DETAILED DESCRIPTION

Certain aspects and features described herein relate to sensor-baseduser detection, identification, and authentication within customerservice locations. In some embodiments, scanning systems includingvarious sensor devices may be used at customer service locations todetect and identify users based on user-specific characteristicsreceived via cameras and other scanning devices installed at thelocation. For example, a scanning system including one or more camerasand other scanning devices may be configured to monitor and capturesensor data from a location. In some implementations, a camera devicemay include a camera and the various sensor devices. Customer servicelocations may include locations such as banks or other financialinstitutions, as well as other retail locations (e.g., stores,restaurants, etc.), educational or governmental offices, and the like.In some embodiments, a scanning system may receive sensor data from itsvarious scanning devices at the location, and may analyze the sensordata to detect a user at the location. For example, image data fromcameras at the location, depth sensors, infrared sensors, door sensors,and various other sensors may be used to detect that a customer or otheruser has entered a particular location. Concurrently with or afterdetecting a user at the location, multiple user characteristics may becollected for the user, including visual characteristics, voice/audiocharacteristics, biometrics, and/or movement characteristics of theuser. The user may be identified based on a combination of these usercharacteristics. In some cases, a data store may store combinations ofuser characteristics (e.g., visual, movement, biometric, etc.)associated with user profiles. After identifying a user to a thresholdlevel of confidence, the corresponding user profile may be retrieved andused to determine one or more customer interaction strategies. Suchcustomer interaction strategies may include recommendations of productsor services, user-specific personality-based customer servicetechniques, or identification of potential threats of violence or fraudfrom customers. The customer interaction strategies then may be providedto a customer service representative to assist the representative ininteracting with the customer.

At certain customer service locations, a transparent panel having one ormore associated or integrated projector components may separate thecustomer area (e.g., bank lobby, product display areas, etc.) from therestricted areas at the location for which access is limited to thecustomer service representatives (e.g., employees). In such cases, anaugmented reality customer interaction system may be implemented,including various cameras, location sensors, and the transparent panelincluding the associated projection components. In some embodiments, theaugmented reality customer interaction system may receive image datacaptured via cameras directed to the customer side of the transparentpanel. The individual users on the customer side of the transparentpanel may be identified based on the image data and/or other sensor datareceived by the system (e.g., biometrics, user access codes, etc.), andassociated user data may be retrieved for the identified users. Forinstance, the system may retrieve user account data, previoustransactions, product or service recommendations, and the like. Customerinteraction data such as a customer service script and/or productrecommendations may be determined based on the retrieved user data, andthe customer interaction data may be projected to the customer servicerepresentative side of the transparent panel. As discussed below in moredetail, the customer interaction data may be projected so that isvisible only on one side of the transparent panel (e.g., the customerservice representative side) and is not visible to customers on theother side of the panel.

In some embodiments, projecting the customer interaction data may beperformed using projectors integrated with the transparent panel.Additionally or alternatively, the data may be projected onto the panelfrom a separate projector, so that the data may be reflected by thepanel and visible to the customer service representative. In variousembodiments, the source from which the data is projected, the targetlocation of the projection, and the angle of the projected data may becalculated based on the positions of the customer and customer servicerepresentative, so that the data appears next to the associated customerwithin the field of vision of the customer service representative. Thus,the projection of the customer interaction data at the appropriate placeon the transparent panel provides an augmented reality user interfacefor the customer service representative, where the customer interactiondata appears next to the associated customer from the perspective of thecustomer service representative.

Thus, the augmented reality customer interaction system may use camerasand sensors to determine and track both the locations of users on oneside of the transparent panel (e.g., customers), and users on the otherside of the transparent panel (e.g., customer service representatives).As users on both sides of the transparent panel move, the system maydetect the movements and updated locations of the users, and then mayupdate the projection of the customer interaction data so that itremains attached to the customer from the perspective of the customerservice representative. In some cases, the system may allow customerservice representatives to edit or update the interaction data forparticular customers, and the updated data may remain associated withthe customer while the customer remains at the location, including forthe customer's interactions with other customer service representatives,and for subsequent visits by the customer to the same location orassociated locations.

In the following description, for the purposes of explanation, specificdetails are set forth in order to provide a thorough understanding ofvarious implementations and examples. Various implementations may bepracticed without these specific details. For example, circuits,systems, algorithms, structures, techniques, networks, processes, andother components may be shown as components in block diagram form inorder not to obscure the implementations in unnecessary detail. Thefigures and description are not intended to be restrictive.

In some examples, each process in the figures of this disclosure can beperformed by one or more processing units. A processing unit may includeone processor or multiple processors, including single core or multicoreprocessors, one or more cores of processors, or combinations thereof. Insome examples, a processing unit can include one or more special purposeco-processors such as graphics processors, Digital Signal Processors(DSPs), or the like. In some examples, some or all of the processingunits can be implemented using customized circuits, such as ApplicationSpecific Integrated Circuits (ASICs), or Field programmable gate arrays(FPGAs). As used herein, reference to a processor may include oneprocessor or more than one processor.

FIG. 1 is a diagram illustrating an example of a customer serviceenvironment 100 according to some aspects of the present disclosure. Thecustomer service environment 100 may include a transparent panel 110separating the users (e.g., customers) 120 b in the customer area 100 bfrom employees (e.g., customer service representatives) 120 a in theemployee area 100 a. In some cases, the customer service environment 100may be a bank or other financial institution, in which the bankemployees (customer service representatives 120 a) are separated fromcustomers 120 b by a secure transparent panel 110. In other cases, thecustomer service environment 100 may be a retail location, such as astore, gas station, or convenience store, or may be an administrativeoffice, governmental office, educational office, etc. In this example,the transparent panel 110 functions as part of an augmented realitycustomer interaction system, in which customers 120 b may be identifiedand customer-specific interaction data 130 may be projected onto (orfrom) a surface of the transparent panel 110 for viewing by the customerservice representatives 120 a.

FIG. 2 is a diagram illustrating another example of a similar oridentical customer service environment at a location 200 according tosome aspects of the present disclosure. The customer service environmentat the location 200, may include a transparent panel 210 separating acustomer service representative 220 a on a restricted side 200 a of thetransparent panel 210 from a customer 220 b on a customer side 200 b ofthe transparent panel 210. As in the previous example, the identity ofthe customer 220 b may be determined based on data gathered by a numberof cameras and other sensors 240 installed at the customer serviceenvironment at the location 200, and one or more projection components250 may be used to provide an augmented reality user interface thatdisplays customer interaction data 230 specific to the customer 220 b tothe customer service representative 220 a. The cameras and sensors 240may include still image and video cameras, audio sensors, motion sensors(e.g., infrared sensors, depth sensors, etc.), biometric sensors (e.g.,fingerprint scanners, retina or iris scanners, etc.), customer inputdevices (e.g., keypads and user interface terminals), customer devicedetectors (e.g., using NFC or Bluetooth to detect a known customermobile device), and other sensors or detector devices capable ofdetecting characteristics of a customer 220 b. As discussed below, invarious examples the cameras and sensors 240 may be used for facialrecognition analysis, ear recognition analysis, biometric analysis, gaitanalysis, user height/weight/size analysis, or to detect otheridentifying characteristics of customers 220 b on the customer side 200b of the transparent panel 210. In some implementations, camera devicesmay include cameras, sensors, and detector devices. One or moreprojection components 250 may be integrated into the transparent panel210, or may be configured to project images onto a surface of thetransparent panel 210, to provide an augmented reality interface withcustomer interaction data 230 specific to the customer 220 b that can beviewed by the customer service representative 220 a.

In various embodiments, the customer interaction systems shown in FIG. 1and FIG. 2 may be augmented reality systems, or “mixed reality” (orpolyplexity) systems which merge real and virtual worlds. Transparentpanel 210, which may be transparent or translucent, may be configured toallow for the injection of augmented reality elements 235 (e.g.,floating windows) into the line-of-sight of customer servicerepresentatives 220 a, without completely obscuring their view of thecustomer 220 b or the customer side 200 b of the customer serviceenvironment at the location 200. As discussed below, the projection ofimages within or onto the transparent panel 210 may be updated inresponse to movement by the customer 220 b or employee ((customerservice representative 220 a), so that the employee (customer servicerepresentative 220 a) may walk around and look at customer 220 b fromdifferent distances and angles while the customer interaction data 230remains “attached” to the customer 220 b from the perspective of theemployee (customer service representative 220 a).

As shown in FIG. 2, a user scanning system and/or augmented realitycustomer interaction system may include a number of cameras and othersensors 240 facing out (or otherwise directed) into the customer side200 b of the location. Cameras and sensors 240 may be configured to scanthe area and collect data from multiple angles. In some embodiments, thesystem may use the data collected by the location cameras and othersensors 240 to generate a digital model of the location, in order todetermine the size, location, angles, and movement of the customerinteraction data 230 to be displayed via the transparent panel 210. Forinstance, the augmented reality (or mixed reality) interface system mayuse a combination of standard video and infrared depth-sensing vision todraw a digital picture of the customer side 200 b of the location 200.

In some embodiments, an augmented reality customer interaction systemsuch as those shown in FIG. 1 and FIG. 2 also may determine the locationof the customer service representative 220 a, including positiontracking as the customer service representative 220 a moves on therestricted side 200 a of the transparent panel 210. In some cases,cameras and other sensors 240 may be directed to the restricted side 200a of the transparent panel 210. The customer 220 b and/or customerservice representative 220 a also may be outfitted with wearable devices(e.g., smartphones, smart watches, wearable computer glasses, etc.) insome cases, having motion-tracking sensors (e.g., accelerometer,gyroscope, magnetometer compass, etc.) to detect the precise positionand movement of the users 220. Specifically, cameras and sensors 240and/or other positioning techniques may provide advantages when used todetermine the position/angle of the head and eyes of the customerservice representative 220 a, in order to allow the system to determinethe position for the customer interaction data 230 on a surface of thetransparent panel 210.

In some embodiments, the transparent panel 210 may be constructed usinglayers of colored glass to allow for the creation of graphical imagesthat the customer service representatives 220 a may interact with, orview from different angles. The transparent panel 210 also may beconstructed as a passthrough device, so that customer servicerepresentatives 220 a may view the real world through the transparentpanel 210, while images (e.g., holograms) may be projected out in frontof the customer service representative 220 a.

In some cases, the transparent panel 210 of the augmented reality systemmay include integrated projectors that may be built directly into thetransparent panel 210. For instance, such projectors may be implementedas using liquid crystal on silicon (LCoS) displays mounted on or nearthe transparent panel 210. These projectors may direct out images, whichthen pass through a combiner that combines the projected images and thereal world. The transparent panel 210 may use total internal reflection(TIR) in some embodiments. TIR may, depending on the shape of the prismused, bounce light internally or aim light toward the eyes of thecustomer service representative 220 a. Waveguides also may be usedwithin the transparent panel 210, and/or a surface coating may beapplied to allows the surface of the transparent panel 210 to create aseries of diffraction gratings. For example, an image to be displayedmay be transmitted through the optics of the transparent panel 210,coupled in through the diffraction grating, and diffracted inside thewaveguide. The image then may be out-coupled. Different types ofgratings may be used to make RGB color holograms, which may beimplemented based on layered plates that form the transparent panel 210.Thus, the transparent panel 210 may project out an image to the customerservice representative 220 a, which then may be combined, diffracted andlayered to produce images visible in space to the customer servicerepresentative 220 a.

In various embodiments, for customer interaction systems using virtualreality, augmented reality, and/or mixed reality, the cameras andsensors 240 and corresponding positioning software may be used toperform head tracking, eye tracking, and depth sensing of the customerservice representative 220 a, as well as detailed room mapping both therestricted side 200 a and customer side 200 b of the transparent panel210. Such cameras and sensors 240 may include cameras positioned atmultiple angles, depth cameras; ambient light sensors, and/or photo/HDvideo cameras. Environmental sensing cameras may be included to providethe data for head tracking. A time of flight (ToF) depth camera may beincluded to serve two roles: to help with hand tracking in embodimentswhere the representative may use their hands to manipulate the customerinteraction data 230 or other augmented reality elements, and also toperform surface reconstruction which may be used to place augmentedreality elements on physical objects such as customers 220 b.

As described above, projector components (e.g., liquid crystalprojectors, combiners, diffraction gratings and waveguides) may beintegrated into the transparent panel 210 to project image data directlyto a customer service representative 220 a. In other cases, externallypositioned projection components 250 may be used, in conjunction with anangled and reflective surface, to project images that will be reflectedback to the customer service representative 220 a at the correct pointwithin the representative's field of vision. The angle and/or layeredcomposition of the transparent panel 210 may allow for the projectedimage to be visible on the restricted side 200 a of the transparentpanel 210 but not on the customer side 200 b.

FIG. 3 is a diagram illustrating an example of a distributed computingenvironment 300 according to some aspects of the present disclosure. Thedistributed computing environment 300 may include a computer server 302,three front-end (or peripheral) computing devices (e.g., local cameras306 a, scanning devices 306 b (e.g., sensors), and a surface projectionsystem 306 c), and other components that may implement certainembodiments and features described herein. In some embodiments, theserver 302 and computing devices may operate at a customer serviceenvironment 100 or 200 (as shown in FIGS. 1 and 2, respectively) toimplement a user scanning system or augmented reality customer interfacesystem as described above. For example, server 302 may be a computerserver operating a bank branch location, retail store location,administrative office location, etc. It may communicate with and controlfront-end peripheral devices such as local cameras 306 a, scanningdevices 306 b (e.g., sensors), and a one-way surface projection system306 c to implement the user scanning capabilities and augmented realitycustomer user interface as described above.

Front-end peripheral devices 306 a-306 c may be, for example, off theshelf computing components configured to operate at a location 200, andcommunicate with and be controlled by server 302. Cameras 306 a maycapture image data at the location 200, scanning devices 306 b mayinclude various other sensors to capture additional data at the location200, and surface projection system 306 c may include a specializedsurface and/or various projection components 250 configured to providean augmented reality or mixed reality interface using the transparentpanel 210. Server 302 may be communicatively coupled with the front-endperipheral devices 306 a-306 c via one or more communication networks.Front-end peripheral devices 306 a-306 c may receive client applicationsfrom server 302 or from other application providers (e.g., public orprivate application stores). Server 302 may be configured to run one ormore server software applications or services, for example, web-based orcloud-based services, to support interaction with front-end peripheraldevices 306 a-306 c. In some implementations, camera devices may includethe sensors and the cameras.

Various different subsystems and/or components 304 may be implemented onserver 302. The subsystems and components within the server 302 andfront-end devices 306 may be implemented in hardware, firmware,software, or combinations thereof. Various different systemconfigurations are possible in different distributed computing systems300. Additionally, although exemplary computing environment 300 is shownwith three front-end peripheral devices 306 a-306 c, any number of suchdevices may be supported. Other devices, such as specialized sensordevices, etc., may interact with front-end peripheral devices 306 a-306c and/or server 302.

Computing environment 300 also may include one or more data stores 310and/or back-end servers 312. For example, data stores 310 may storecustomer images and other characteristic data that may be used toidentify customers detected at a location 200 based on the datacollected by the front-end peripheral devices such as cameras 306 a andscanning devices 306 b (e.g., sensors). Data stores 310 also may storecustomer profiles, account data, previous customer interaction data,etc. Back-end servers 312 may include, for example, productrecommendation engines, fraud/risk detection engines, etc., which may beused to generate the specific customer interaction data 230 once acustomer has been identified. Data stores 310 and back-end servers 312may reside in the same datacenter or may operate at a remote locationfrom server 302. In some cases, one or more data stores 310 may resideon a non-transitory storage medium within the server 302. Other datastores 310 and back-end servers 312 may be remote from server 302 andconfigured to communicate with server 302 via one or more networks 320.In certain embodiments, data stores 310 and back-end servers 312 mayreside in a storage-area network (SAN), or may use storage-as-a-service(STaaS) architectural model.

As shown, server 302 may use various security and integration components308 to transmit, receive, and manage communications between the server302 and the back-end data stores 310 and back-end severs 312. Althoughnot shown in this example, server 302 also may use similar componentsand additional networks to communicate with and control the front-endperipheral devices 306 a-306 c, for example, when the server 302 islocated remotely from the front-end peripheral devices 306 a-306 c. Thesecurity and integration components 308 may include separate servers,such as web servers and/or authentication servers, and/or specializednetworking components, such as firewalls, routers, gateways, loadbalancers, and the like. For example, the security and integrationcomponents 308 may include one or more dedicated web servers and networkhardware in a datacenter or a cloud infrastructure. In other examples,the security and integration components 308 may correspond to separatehardware and software components which may be operated at a separatephysical location and/or by a separate entity.

Security and integration components 308 may implement various securityfeatures for data transmission and storage, such as authenticating usersand restricting access to unknown or unauthorized users. In variousimplementations, security and integration components 308 may provide,for example, a file-based integration scheme or a service-basedintegration scheme for transmitting data between a particular location200 (e.g., bank branch, retail location, office, etc.) and a centralizedback-end datacenter and network of computing systems associated with thelocation 200. Security and integration components 308 may use securedata transmission protocols and/or encryption for data transfers, forexample, File Transfer Protocol (FTP), Secure File Transfer Protocol(SFTP), and/or Pretty Good Privacy (PGP) encryption. In someembodiments, one or more web services may be implemented within thesecurity and integration components 308, including cross-domain and/orcross-platform web services, which may be developed for enterprise usein accordance with various web service standards, such as RESTful webservices (i.e., services based on the Representation State Transfer(REST) architectural style and constraints), and/or web servicesdesigned in accordance with the Web Service Interoperability (WS-I)guidelines. Some web services may use the Secure Sockets Layer (SSL) orTransport Layer Security (TLS) protocol to provide secure connectionsbetween the server 302 and back-end servers 312 or back-end data stores310. SSL or TLS may use HTTP or HTTPS to provide authentication andconfidentiality. In other examples, web services may be implementedusing REST over HTTPS with the OAuth open standard for authentication,or using the WS-Security standard which provides for secure SOAPmessages using XML encryption. In other examples, the security andintegration components 308 may include specialized hardware forproviding secure web services. For example, security and integrationcomponents 308 may include secure network appliances having built-infeatures such as hardware-accelerated SSL and HTTPS, WS-Security, andfirewalls. Such specialized hardware may be installed and configured infront of any web servers, so that any external devices may communicatedirectly with the specialized hardware.

Communication network(s) 320 may be any type of network familiar tothose skilled in the art that can support data communications using anyof a variety of commercially-available protocols, including withoutlimitation, TCP/IP (transmission control protocol/Internet protocol),SNA (systems network architecture), IPX (Internet packet exchange),Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols,Hyper Text Transfer Protocol (HTTP) and Secure Hyper Text TransferProtocol (HTTPS), Bluetooth®, Near Field Communication (NFC), and thelike. Merely by way of example, network(s) 320 may be local areanetworks (LAN), such as one based on Ethernet, Token-Ring and/or thelike. Network(s) 320 also may be wide-area networks, such as theInternet. Networks 320 may include telecommunication networks such as apublic switched telephone networks (PSTNs), or virtual networks such asan intranet or an extranet. Infrared and wireless networks (e.g., usingthe Institute of Electrical and Electronics (IEEE) 802.11 protocol suiteor other wireless protocols) also may be included in networks 320.

FIG. 4 is a block diagram illustrating an example embodiment of acomputer server 302 according to some aspects of the present disclosure.The computer server 302 may be used to implement the user scanningsystems and/or augmented reality customer interface systems describedherein. Server(s) 302 may include various server hardware and softwarecomponents that manage the front-end devices 306, collect and analyzedata from cameras 306 a and scanning devices 306 b, and provideinteractive and adaptive content to users via one or more surfaceprojection systems 306 c.

For example, server 302 may include a user identification system 402.The user identification system 402 may be implemented using dedicated orshared hardware and software resources within the server 302. The useridentification system 402 may be configured to request and retrieveimage data and other sensor data from a location 200 at which users(e.g., customers) may be present. The user identification system 402 maydynamically control cameras 306 a and other scanning devices 306 b(e.g., movement sensors, biometric sensors, user interface systems,etc.) to collect user characteristics for a number of users present atthe location 200. In some implementations, camera devices may includethe sensors and the cameras. The user characteristic data may beanalyzed and compared to previously stored user characteristic data toidentify users as existing customers or other known individuals. In somecases, a combination of multiple different user characteristic data maybe matched to previously stored data which may reside locally within theserver 302 or remotely within a back-end system (e.g., data stores 310or back-end servers 312). The user identification system 402 also mayimplement a confidence threshold for identifying users, and may controlcameras 306 a and scanning devices 306 b to collect and processadditional user characteristic data as the user moves around within thecustomer side 200 b of the location 200, until reaching a minimumconfidence threshold that the user can be positively identified. Theconfidence threshold may be based on the confidence of the individualmatching factors (e.g., the confidence level of a particular facialrecognition match), the reliability of the different types of usercharacteristic matching processes (e.g., fingerprint matching may bemore reliable that facial recognition, which may be more reliable thangait recognition, etc.), and the number of different matching factorsfor which a positive user match was identified.

The server 302 also may include a recommendation engine 404. Asdescribed below, based on the user profiles and previous customerinteractions for the users identified within the customer side 200 b ofthe location 200, user-specific recommendations for customer interactionstrategies may be determined by the recommendation engine 404. Suchcustomer interaction strategies may include recommendations for productsor services, customer communication preferences (e.g., languages), orrecommendations based on customer personality traits or previouspositive or negative interactions with the customer. The recommendationengine 404 may be implemented entirely at the server 302, or maycommunicate with back-end systems (e.g., data stores 310 or back-endservers 312) to retrieve and analyze the customer's account data,previous customer interactions, etc.

The server 302 also may include an augmented reality (or mixed reality)projection system 406. The augmented reality projection system 406 maybe implemented using dedicated hardware at the location 200 (e.g.,projection components 250 and transparent panel 210) as described above.After determining a recommendation for customer interaction strategy fora particular customer 220 b, including an interaction script,recommendations, customer information, etc., the augmented realityprojection system 406 may be configured to generate an augmented realityuser interface for one or more customer service representatives 220 athat may interact with the customer 220 b. For instance, usingprojection components 250 and transparent panel 210, the augmentedreality projection system 406 may generate and project customerinteraction data to a particular location on a surface of thetransparent panel 210 so that it may be visible to a customer servicerepresentative 220 a and not to the customer 220 b. The projectionsystem 406 may, for example, determine and track the positions of thecustomer service representative 220 a and the customer 220 b, and mayproject the customer interaction data 230 to a position on thetransparent panel 210 so that the customer interaction data 230 will beassociated with the customer 220 b from the visual perspective of thecustomer service representative 220 a. For example, the customerinteraction data 230 may be attached to the customer 220 b from theperspective of the customer service representative 220 a, and theprojection system 406 may move the customer interaction data 230 asneeded to maintain the attachment of the augmented reality data to thecustomer 220 b as both the customer 220 b and the customer servicerepresentative 220 a may move around on their respective sides of thetransparent panel 210.

The server 302 also may include a user tracking and monitoring system408. The user tracking and monitoring system 408 may track the locationsof multiple customers 220 b and multiple customer servicerepresentatives 220 a at the location, in order to assign specificcustomers to specific representatives and manage the overall customerexperience at the location 200. For example, in scenarios where multiplecustomers 220 b at the location 200 and/or multiple customer servicerepresentatives 220 a are available to interact with the customers 220b, the user tracking and monitoring system 408 may determine a preferredcustomer service representative 220 a to interact with a particularcustomer 220 b. Such determinations may be based on customer languagepreferences (e.g., a Spanish-speaking customer may be routed toparticular representative fluent in Spanish), customer accounttype/details (e.g., a customer with a large or complex account portfoliomay be routed to a senior service representative), or customertemperament (e.g., a difficult customer with a history of negativeinteractions may be routed to a manager), and so on. Such routing may beperformed by means of the projection system 406. For example, thecustomer interaction data 230 for a customer 220 b may be projected sothat it is visible to the preferred customer service representative 220a for that customer.

The user tracking and monitoring system 408 also may allow customerservice representatives 220 a to update the customer interaction data230 during the customer's visit to the location 200. For instance, if acustomer 220 b interacts with a first customer service representative220 a to request certain information or attempt to perform a certaintype of purchase or transaction, the first customer servicerepresentative 220 a may update the customer interaction data 230 forthat customer 220 b to indicate the customer's purpose at the location200, the customer's current mood, and/or other customer information.Then, if the same customer 220 b interacts with a different customerservice representative 220 a during the same visit to the location, orduring a subsequent visit to the same or different related location,then the customer interaction data 230 for that customer 220 b mayautomatically include the customer's purpose, questions, mood, or anyother customer information input by the first customer servicerepresentative, which may benefit the second customer servicerepresentative 220 a in assisting the customer 220 b.

FIG. 5 is a block diagram of an illustrative computer system 500according to some aspects of the present disclosure. The computer system500 may correspond to the computer server 302, front-end peripheraldevices 306 a-306 c, or any other of the computing devices or serversdescribed herein. In this example, computer system 500 includesprocessing units 504 that communicate with a number of peripheralsubsystems via a bus subsystem 502. These peripheral subsystems include,for example, a storage subsystem 510, an I/O subsystem 526, and acommunications subsystem 532.

Bus subsystem 502 provides a mechanism for enabling the variouscomponents and subsystems of computer system 500 to communicate witheach other. Although bus subsystem 502 is shown schematically as asingle bus, alternative embodiments of the bus subsystem may utilizemultiple buses. Bus subsystem 502 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Sucharchitectures may include, for example, an Industry StandardArchitecture (ISA) bus, Micro Channel Architecture (MCA) bus, EnhancedISA (EISA) bus, Video Electronics Standards Association (VESA) localbus, and Peripheral Component Interconnect (PCI) bus, which can beimplemented as a Mezzanine bus manufactured to the IEEE P1386.1standard.

Processing unit 504, which may be implemented as one or more integratedcircuits (e.g., a conventional microprocessor or microcontroller),controls the operation of computer system 500. One or more processors,including single core and/or multicore processors, may be included inprocessing unit 504. As shown in the figure, processing unit 504 may beimplemented as one or more independent processing units 506 and/or 508with single or multicore processors and processor caches included ineach processing unit. In other embodiments, processing unit 504 may alsobe implemented as a quad-core processing unit or larger multicoredesigns (e.g., hexa-core processors, octo-core processors, ten-coreprocessors, or greater.

Processing unit 504 may execute a variety of software processes embodiedin program code (e.g., such as software processes corresponding tosystems 402-408), and may maintain multiple concurrently executingprograms or processes. At any given time, some or all of the programcode to be executed can be resident in processing unit(s) 504 and/or instorage subsystem 510. In some embodiments, computer system 500 mayinclude one or more specialized processors, such as digital signalprocessors (DSPs), outboard processors, graphics processors,application-specific processors, and/or the like.

Input/Output (I/O) subsystem 526 may include device controllers 528 forone or more user interface input devices and/or user interface outputdevices 530. User interface input and output devices 530 may be integralwith the computer system 500 (e.g., integrated audio/video systems,and/or touchscreen displays), or may be separate peripheral deviceswhich are attachable/detachable from the computer system 500.

Input devices of the user I/O devices 530 may include a keyboard,pointing devices such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, adial, a button, a switch, a keypad, audio input devices with voicecommand recognition systems, microphones, and other types of inputdevices. Input devices of the user I/O devices 530 may also includethree dimensional (3D) mice, joysticks or pointing sticks, gamepads andgraphic tablets, and audio/visual devices such as speakers, digitalcameras, digital camcorders, portable media players, webcams, imagescanners, fingerprint scanners, barcode readers, 3D scanners, 3Dprinters, laser rangefinders, and eye gaze tracking devices. Additionalinput devices of the user I/O devices 530 may include, for example,motion sensing and/or gesture recognition devices that enable users tocontrol and interact with an input device through a natural userinterface using gestures and spoken commands, eye gesture recognitiondevices that detect eye activity from users and transform the eyegestures as input into an input device, voice recognition sensingdevices that enable users to interact with voice recognition systemsthrough voice commands, medical imaging input devices, MIDI keyboards,digital musical instruments, and the like.

Output devices of the user I/O devices 530 may include one or moredisplay subsystems, indicator lights, or non-visual displays such asaudio output devices, etc. Display subsystems may include, for example,cathode ray tube (CRT) displays, flat-panel devices, such as those usinga liquid crystal display (LCD) or plasma display, light-emitting diode(LED) displays, projection devices, touch screens, and the like. Ingeneral, use of the term “output device” is intended to include allpossible types of devices and mechanisms for outputting information fromcomputer system 500 to a user or other computer. For example, outputdevices of the user I/O devices 530 may include, without limitation, avariety of display devices that visually convey text, graphics andaudio/video information such as monitors, printers, speakers,headphones, automotive navigation systems, plotters, voice outputdevices, and modems.

Computer system 500 may comprise one or more storage subsystems 510,comprising hardware and software components used for storing data andprogram instructions, such as system memory 518 and computer-readablestorage media 516. The system memory 518 and/or computer-readablestorage media 516 may store program instructions that are loadable andexecutable on processing units 504, as well as data generated during theexecution of these programs.

Depending on the configuration and type of computer system 500, systemmemory 318 may be stored in volatile memory (such as random accessmemory (RAM) 512) and/or in non-volatile storage drives 514 (such asread-only memory (ROM), flash memory, etc.) The RAM 512 may contain dataand/or program modules that are immediately accessible to and/orpresently being operated and executed by processing units 504. In someimplementations, system memory 518 may include multiple different typesof memory, such as static random access memory (SRAM) or dynamic randomaccess memory (DRAM). In some implementations, a basic input/outputsystem (BIOS), containing the basic routines that help to transferinformation between elements within computer system 500, such as duringstart-up, may typically be stored in the non-volatile storage drives514. By way of example, and not limitation, system memory 518 mayinclude application programs 520, such as client applications, Webbrowsers, mid-tier applications, server applications, etc., program data522, and an operating system 524.

Storage subsystem 510 also may provide one or more tangiblecomputer-readable storage media 516 for storing the basic programmingand data constructs that provide the functionality of some embodiments.Software (programs, code modules, instructions) that when executed by aprocessor provide the functionality described herein may be stored instorage subsystem 510. These software modules or instructions may beexecuted by processing units 504. Storage subsystem 510 may also providea repository for storing data.

Storage subsystem 510 may also include a computer-readable storage mediareader (not shown) that can be connected to computer-readable storagemedia 516. Together and, optionally, in combination with system memory518, computer-readable storage media 516 may comprehensively representremote, local, fixed, and/or removable storage devices plus storagemedia for temporarily and/or more permanently containing, storing,transmitting, and retrieving computer-readable information.

Computer-readable storage media 516 containing program code, or portionsof program code, may include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to, volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information. This can include tangible computer-readable storagemedia such as RAM, ROM, electronically erasable programmable ROM(EEPROM), flash memory or other memory technology, CD-ROM, digitalversatile disk (DVD), or other optical storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or other tangible computer readable media. This can also includenontangible computer-readable media, such as data signals, datatransmissions, or any other medium which can be used to transmit thedesired information and which can be accessed by computer system 500.

By way of example, computer-readable storage media 516 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 516 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 516 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 500.

Communications subsystem 532 may provide a communication interface fromcomputer system 500 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 5, thecommunications subsystem 532 may include, for example, one or morenetwork interface controllers (NICs) 534, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 536, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 532 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, FireWire® interfaces, USB® interfaces, and the like.Communications subsystem 536 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

The various physical components of the communications subsystem 532 maybe detachable components coupled to the computer system 500 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 500. Communicationssubsystem 532 also may be implemented in whole or in part by software.

In some embodiments, communications subsystem 532 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 500. For example,communications subsystem 532 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators 309). Additionally, communications subsystem 532may be configured to receive data in the form of continuous datastreams, which may include event streams of real-time events and/orevent updates (e.g., sensor data applications, financial tickers,network performance measuring tools, clickstream analysis tools,automobile traffic monitoring, etc.). Communications subsystem 532 mayoutput such structured and/or unstructured data feeds, event streams,event updates, and the like to one or more data stores 104 that may bein communication with one or more streaming data source computerscoupled to computer system 500.

Due to the ever-changing nature of computers and networks, thedescription of computer system 500 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

FIG. 6 is a flowchart illustrating an example of a process 600 foridentifying a user at a location based on multiple sensors andmulti-factor user characteristic matching according to some aspects ofthe present disclosure. As described below, the techniques in FIG. 6 maybe used to detect and identify a customer 220 b within a location 200,such as a bank, retail location, office, etc. Multiple differentsensors, detectors, and location scanning devices may be used toidentify various characteristics of the user, including visualcharacteristics, biometric characteristics, movement characteristics,and other characteristics of the user. In some implementations, cameradevices may include the sensors and the cameras. The data from multiplesensor devices may be synchronized for time and location to determinemultiple different characteristics of a single user within the location.A particular customer or other known user may be identified based on amulti-factor matching. Multi-factor matching is matching of multipledifferent characteristics of the same user. Based on the identificationof the existing customer or other known user within a confidencethreshold, a user profile or other user data may be retrieved and aninteraction strategy may be determined for handling customer service orother interactions with the user.

In blocks 601-603, the server 302 may receive location sensor data fromvarious different sensor devices. Although blocks 601-603 describe thatsensor data may be received from three different sensor devices, invarious examples sensor data may be received from any number of cameras306 a or other scanning devices 306 b (e.g., sensors). The sensor datareceived in blocks 601-603 may include image or streams of video datafrom multiple cameras 306 a operating on the customer side 200 b of alocation 200. The image or video data captured from cameras 306 a may beused for facial recognition analysis, ear recognition analysis, usersize height or size analysis, gait or movement analysis, and the like.Different cameras 306 a may be positioned differently within thelocation 200 to capture image or video data of users from differentheights and at different angles.

In addition to image and video data received from cameras 306 a, variousother data may be received from scanning devices 306 b in blocks601-603. Such sensor data may include biometric data collected from theusers on the customer side 200 b of a location 200. Examples ofbiometric data that may be collected may include fingerprint data from afingerprint scanner at the location 200, voice data from one or moremicrophones at the location 200, retina or iris scans from scanners atthe location 200, signature data collected from the user at thelocation, and so on. In some implementations, camera devices may includethe sensors and the cameras.

In some embodiments, scanning devices 306 b may include devicesconfigured to detect and identify one or more mobile devices worn by orcarried by users within the location, such as smartphones, smartwatches, computer glasses, etc. Scanning devices 306 b may automaticallycommunicate with the customer's mobile devices via a wireless protocolsuch as Bluetooth or NFC, to identify a device name, device type, MACaddress, or other identifying information of the device. Such deviceidentifying information may be used in conjunction with user image dataand/or other sensor data to identify a customer or other known user.

Additionally, scanning devices 306 b at the location may include userterminals, keypads, card readers, or other devices configured tointeract with the user. For example, in certain secure locations 200,users may be required to insert a card or scan a fob or otheridentification data before they are permitted access to the location200. In other examples, a user at the location may optionally interactwith a check-in kiosk or terminal, or may be required to interact withsuch a kiosk or terminal, to provide their name or other identifyinginformation and the purpose of their visit. This additionaluser-identifying information also may be used in conjunction with userimage data and/or other sensor data to identify customers or other knownusers at the location.

In block 604, the server 302 may analyze the data received from thecameras 306 a and scanning devices 306 b, to determine and isolate theparticular characteristics of users at the location 200. In some cases,the server 302 may preprocess and transmit sensor data (e.g., facialimages, biometric data, etc.) to one or more backend systems 310-312 todetermine whether or not the received data matches a known user.Additionally or alternatively, the server 302 may analyze the image andsensor data locally, and compare the data to locally stored usercharacteristic data (e.g., frequent customers or visitors to a location200, such as a specific branch location), to identify a user matchingthe image data and/or sensor data.

The analysis performed in block 604 may include initially synchronizingthe image data and sensor data from multiple cameras 306 a and scanningdevices 306 b, with respect to time and user position on the customerside 200 b of the location 200. As noted above, the customer side 200 bof the location 200 may include multiple different customers at once.Additionally, the different customers at the location 200 may enter andleave the location 200 periodically, and may move around and mingle withother customers in the location 200. Thus, the server 302 may beconfigured to synchronize the image and sensor data from multiplecameras 306 a and scanning devices 306 b, so that data from multiplesources can be collected and evaluated for a single user. The cameras306 a and scanning devices 306 b detecting the user data, as well as theserver 302 receiving and processing the user data, also may beconfigured to distinguish between different users in the location 200,to track individual users and they move around within the customer side200 b of the location 200, and to retain user characteristic data tomatch customers as they exit and re-enter the location 200.

After time synchronizing and position synchronizing the data received inblocks 601-603 to isolate a single user within the location 200, atblock 604, analysis of the data may be performed. The analysis performedin block 604 may include various different types of user identificationanalyses. For instance, the server 302 may perform (e.g., individuallyor via requests to back-end servers 312 and back-end data stores 310)facial recognition analysis, ear recognition analysis, biometricanalysis, voiceprint analysis, written signature analysis, gaitanalysis, user height/weight/size analysis, among other useridentification techniques described herein. FIGS. 7A and 7B illustrateexamples of collecting visual facial and ear recognition data accordingto aspects of the present disclosure. Referring to FIG. 7A visual facialrecognition data may be collected on a first user via a first set ofcameras 306 a. As shown in FIG. 7B, ear recognition data may becollected on a second user via a second set of cameras 306 a operatingat the location 200. In some implementations, both facial recognitionand ear recognition may be performed on the same user.

Referring again to FIG. 6, in block 605, the server 302 may perform amulti-factor user characteristic match to determine if a customer orother known user may be identified based on the combination of the imagedata and/or sensor data. As noted above, a multi-factor usercharacteristic matching process may include matching multiple differentcharacteristics of a single user received from different cameras 306 aand/or scanning devices 306 b at the location 200. Multi-factor usercharacteristic matching may be used in some implementations, and mayprovide advantages of additional user authentication and security toprevent misidentification of a user by a single system, for example,misidentification of a bad actor attempting to impersonate a differentuser. In some embodiments, the server 302 may implement a two-factoruser characteristic matching process in which different useridentification matches may be required to sufficiently identify a userin block 605. A two-factor user characteristic matching process mayinclude, for example, a positive facial recognition match with a userand a positive ear recognition match with the same user. In variousother examples, a two-factor user characteristic matching process mayinclude any combination of facial recognition, ear recognition, gaitrecognition, user size/shape matching, fingerprint matching, voiceanalysis matching, retinal or iris scan matching, and/or any combinationof positive matches of the various user characteristics and sensor datadescribed herein. In certain embodiments, the server 302 may implement athree-factor user characteristic matching process in which threeseparate user characteristic matches may be independent determined, or afour-factor process, a five-factor process, and so on.

If an existing customer or other known user is identified based on themulti-factor user characteristic matching process in block 605, then inblock 606 the server 302 may retrieve a user profile and additional userdata for the identified user. For instance, the server 302 may requestand receive the users account data, profile data, current accountbalances, previous purchases or transaction history data, and the like,from back-end systems (e.g., data stores 310 and back-end servers 312).In block 607, the server 302 may determine a user interaction strategybased on the user data retrieved in block 606. The user interactionstrategy recommendations may provide products or services for thecustomer, user-specific customer language preferences, user-specificpersonality-based customer service techniques, and/or identification ofpotential threats of violence or fraud from the user. As discussedbelow, the user interaction strategies may be provided to theappropriate customer service representative 220 a to assist therepresentative in interacting with the customer 220 b.

In some embodiments, if an existing customer or other known user cannotbe identified based on the multi-factor user characteristic matchingprocess in block 605, then the server 302 may continue monitoring theuser within the location using the same sets of cameras 306 a andscanning devices 306 b, and/or additional front-end peripheral devicesuntil the identity of the user can be determined. For instance, theserver 302 may be initially unable to identify a user that recentlyentered a location 200 such as bank branch or retail store. However, asthe user moves spends more time in the location 200, moves around thelocation, and interacts with people or devices at the location 200,additional image and sensor data may be collected for the user and asubsequent multi-factor user characteristic matching process may besuccessful to identify the user. In some cases, an overall confidencethreshold may be implemented in block 605, where the overall confidenceof the multi-factor match may be based on the confidence of theindividual matching factors (e.g., the confidence level of a particularfacial recognition match), the reliability of the different types ofuser characteristic matching processes (e.g., fingerprint matching maybe more reliable that facial recognition, which may be more reliablethan gait recognition, etc.), and the number of different matchingfactors for which a positive user match was identified.

The specific operations illustrated in FIG. 6 provide a particularprocess for identifying a user at a location based on multiple sensorsand multi-factor user characteristic matching according to anembodiment. Other sequences of operations may also be performedaccording to alternative embodiments. For example, alternativeembodiments may perform the operations outlined above in a differentorder. Moreover, the individual operations illustrated in FIG. 6 mayinclude multiple sub-operations that may be performed in varioussequences as appropriate to the individual operations. Furthermore,additional operations may be added or removed depending on theparticular applications.

FIG. 8 is a flowchart illustrating an example of a process 800 forgenerating and providing augmented reality content (and/or mixed realitycontent) to one side of a transparent panel. The augmented realitycontent (and/or mixed reality content) may be generated to one side of atransparent panel in order to implement a customer interaction strategy.As described below, the techniques in FIG. 8 may be used to provide acustomer service representative 220 a with customer interaction data 230specific to the customer 220 b that may assist the customer servicerepresentative 220 a in interacting with the customer 220 b. Thecustomer interaction data 230 may be provided using a projection systemincluding projection components 250 and a transparent panel 210 in sucha way that the customer interaction data 230 may be attached to thecustomer 220 b from the visual perspective of the customer servicerepresentative 220 a, but also in a way so that it is not visible to thecustomer 220 b. As described below, such augmented reality techniquesmay allow customer service representatives 220 a to be provided withcustomer information, customer interaction scripts, suggestions forproduct or service recommendations, conflict resolution strategies, andother data to allow the customer service representatives 220 a to moreeffectively interact with customers 220 b.

In block 801, a user may be identified at a customer interactionlocation. As described above, the customer interaction location 200 mayinclude a bank, retail location, administrative office, or any locationmonitored by one or more cameras 306 a or scanning devices 306 b. Theidentification of the user may be based on the cameras 306 a andscanning devices 306 b, using the various techniques described above inFIG. 6. In other examples, the identification of the user in block 801may be based on a user data received via a sign-in terminal or kiosk, orbased on the credentials provided by the user to access the location200.

As described above, the customer interaction location may correspond tocustomer service locations at which one or more customer servicerepresentatives 220 a may interact with customers 220 b or other users.In some embodiments, such locations 200 may include a transparent panel210 that separates a customer area on the customer side 200 b of thetransparent panel 210 (e.g., bank lobby, product display areas, etc.)from a restricted area on the restricted side 200 a of the transparentpanel 210 for which access is limited to employees or customer servicerepresentatives. The transparent panel 210 may be part of an augmentedreality projection system, in which projection components 250 may beused to project data onto the transparent panel which may be viewableonly from the area on the restricted side 200 a of the transparent panel210.

In block 802, the server 302 may retrieve customer interaction databased on the user identified in block 801. In some cases, the customerinteraction data may include customer data (e.g., the customer's name,account types, account balances, home city or home branch location,and/or customer personal data), to allow the customer servicerepresentative 220 a to interact more readily with the customer 220 b.Additionally or alternatively, the customer interaction data may includecustomer language preferences, produce or service recommendations, theresults of data analysis regarding the customer's likely purpose of thevisit, or customer management or conflict resolution strategies forinteracting with the customer based on previous interactions with thesame customer.

In block 803, the server 302 may determine the current positions of oneor both of the customer 220 b for which the customer interaction datawas retrieved in block 802, and a customer service representative 220 athat may interact with that customer 220 b. In some cases, a customerservice representative 220 a may be selected from multiple possiblecustomer service representatives 220 a that may be available to interactwith the customer 220 b. For instance, a particular customer servicerepresentative 220 a on the restricted side 200 a of the transparentpanel 210 may be selected based on proximity to the customer 220 b,current availability of the customer service representative 220 a,language compatibility between the customer service representative 220 aand customer 220 b, personality compatibility between the customerservice representative 220 a and customer 220 b, compatibility betweenthe qualifications or expertise of the customer service representative220 a and likely requests of the customer 220 b, or any combination ofthese factors.

After selecting a particular customer service representative 220 a tointeract with the customer 220 b, the server 302 may determine a currentlocation of the customer service representative 220 a and/or thecustomer 220 b in order to position the graphical elements of theaugmented reality user interface. In some embodiments, the server 302may assume the position the customer service representative 220 a and/orthe customer 220 b based on the position of a customer service window,the location of speaking hole at the window, the layout of the desks orcountertops at which the parties are sitting or standing, etc. In otherembodiments, both the positions of the customer service representative220 a and/or the customer 220 b may be determined in block 803 based oncameras 306 a and scanning devices 306 b operating at the location 200.For example, cameras 306 a and scanning devices 306 b directed to therestricted area on the restricted side 200 a of the transparent panel210 may be used to detect the position and height of the customerservice representative 220 a, and may perform head tracking, eyetracking, and depth sensing for precise positioning of the customerservice representative 220 a. Additional cameras 306 a and scanningdevices 306 b directed to the customer side 200 b of the location 200may be used to track the current position, posture, and movement of thecustomer 220 b within the location 200.

In block 804, the server 302 may generate the augmented reality contentand engage the transparent panel 210 and/or projection components 250 toproject the augmented reality content to be visible to the customerservice representative 220 a. The augmented reality content generated inblock 804 may be based on user-specific customer interaction dataretrieved in block 802. As discussed above, the data may include acustomer data, a customer service script, product or servicerecommendations, and other customer-specific information. In block 804,the server 302 may determine the size and positioning of one or moreaugmented reality elements to contain some or all of the customerinteraction data. In some embodiments, the size and positioning ofaugmented reality elements 235 may be determined so that the elementsappear near (or attached to) the associated customer 220 b when viewedfrom the perspective of the customer service representative 220 a. Thus,multiple different augmented reality elements may be generated andprojected to the customer service representative 220 a, where differentelements may be associated with and may appear near or attached todifferent customers 220 b in the location 200. In some cases, the server302 also may select the sizes and positions of the augmented realityelements so as not to obscure the associated customer 220 b or the facesof other users on the customer side 200 b of the location 200, from thevisual perspective of the customer service representative 220 a.

Referring to the example shown in FIG. 2, the augmented reality element235 is shown as a text box projected from the transparent panel 210 toappear as a floating window above the associated customer 220 b. Suchtypes of augmented reality elements 235 may contain customerinformation, scripts, recommendations, and other customer-specificinformation. However, it should be understood that in other examples,any number of different types of augmented reality elements may be used.For instance, different customers 220 b may be labeled and/orcolor-coded using augmented reality elements to identify known andunknown customers, or based on account types and balances, arrival timesand wait times at the location 200, etc. In some cases, customers 220 bthat may be identified as potential fraudsters or may representpotential threats of violence may be especially highlighted to all ofthe customer service representatives 220 a. Additionally, in such cases,the augmented reality elements projected to the customer servicerepresentatives 220 a may include images or video feeds showing thecustomer 220 b from different angles to allow the customer servicerepresentatives 220 a to watch the customer's hand movements, see whatis behind the customer's back, etc.

As described above, in some embodiments, the transparent panel 210 mayinclude integrated projector components such as liquid crystalprojectors, combiners, diffraction gratings, and waveguides to directimages to customer service representatives 220 a on the restricted side200 a of the transparent panel 210. In other examples, externalprojection components 250 may project images that are reflected off oftransparent panel into the field of the vision of a selected customerservice representative 220 a. In block 804, depending on the particularimplementation of the projection components 250 and transparent panel210, the projection system may be engaged to project the augmentedreality elements from or to the appropriate location on the transparentpanel 210.

In block 805, after the initial projection of the augmented realityelements containing the customer interaction data, in such a way to bevisible to the customer service representative 220 a interacting withthe customer 220 b, the server 302 may continue to track the movement ofboth the customer 220 b and the customer service representative 220 athroughout the course of their interaction. User movements may bedetected using cameras 306 a and other scanning devices 306 b at thelocation 200, including detecting where a customer 220 b moves aroundwithin the customer side of the location 200, when the customer 220 band/or customer service representative 220 a switch customer servicewindows/locations, when the customer 220 b and/or customer servicerepresentative 220 a sit down, stand up, or shift from left-to-right orright-to left, and so on. In block 806, in response to detectedmovements by the customer 220 b and/or the customer servicerepresentative 220 a, the server 302 may update the projection of theaugmented reality elements. For example, the server 302 may resize andre-position the augmented reality elements 235 so that the augmentedreality elements 235 can remain “attached to” the customer 220 b fromthe visual perspective of the customer service representative 220 a.Referring briefly to FIG. 9, an example is shown of an augmented realitycustomer user interface that may be provided to a customer servicerepresentative according to some aspects of the present disclosure. Inthis example, customer service representative is viewing a customer 920b through a transparent panel 910, and the server 302 is projecting anaugmented reality element 930 above the customer 920 b from the visualperspective of the representative. As shown in this example, as thecustomer 920 b moves around within the customer service location 900,the server 302 updates the size and positioning of the augmented realityelement 930 so it remains above the customer 920 b from the perspectiveof the representative. Additionally, as shown in this example, as thecustomer 920 b moves around within the customer service location 900,the server 302 may collect additional data from cameras 306 a andscanning devices 306 b at the customer service location 900, which mayincrease the confidence level of the user identification.

In other examples, the server 302 may determine that the interactionbetween the customer service representative 220 a and the customer 220 bhas ended, and may stop projecting the augmented reality elements 235associated with that customer 220 b. Similarly, the server 302 maydetermine that the customer service representative 220 a has handed offthe customer 220 b to a different customer service representative 220 aand is now interacting with a new customer 220 b. In such cases, theserver 302 may update the projections of the augmented reality elementsso that each customer service representative 220 a is able to see thecustomer interaction data 230 for the customers 220 b that the customerservice representative 220 a is currently interacting with.Additionally, in some embodiments, customer service representatives 220a may update the customer interaction data that is visible via theaugmented reality elements 235 during the course of the customer's visitto the location 200. For instance, a representative 220 a may update thecustomer's interaction data 230 based on an initial interaction with thecustomer 220 b to indicate the customer's purpose at the location 200,the customer's current mood, and/or other customer information. Then,when the customer interacts with a second customer servicerepresentative 220 a, during the same visit to the location 200 (orduring a subsequent visit to the same or different locations), then theupdated customer interaction data 230 that may be viewed by the secondcustomer service representative may include the updated data added bythe first representative regarding the customer's purpose, questionsthat the customer asked, the customer's mood, or any other customerinformation that may benefit the second customer service representative220 a in assisting the customer 220 b.

The specific operations illustrated in FIG. 8 provide a particularprocess for generating and providing augmented reality content accordingto an embodiment. Other sequences of operations may also be performedaccording to alternative embodiments. For example, alternativeembodiments may perform the operations outlined above in a differentorder. Moreover, the individual operations illustrated in FIG. 8 mayinclude multiple sub-operations that may be performed in varioussequences as appropriate to the individual operations. Furthermore,additional operations may be added or removed depending on theparticular applications.

Although the subject matter has been described in language specific tostructural features or methodological acts, it is to be understood thatthe subject matter defined in the appended claims is not necessarilylimited to the specific features or acts described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims.

What is claimed is:
 1. A method for providing augmented reality content,the method comprising: identifying a customer at a customer interactionlocation; retrieving customer-specific interaction data; determininglocations of a customer service representative in an area adjacent to afirst surface of a transparent panel and the customer in an areaadjacent to a second surface of the transparent panel; and projecting anaugmented reality element to the first surface of a transparent panel,the augmented reality element displaying the customer-specificinteraction data on the first surface of the transparent panel.
 2. Themethod of claim 1, further comprising: receiving sensor datacorresponding to the customer interaction location from a camera device,the sensor data including visual data; analyzing the sensor data todetect a customer within a proximity of the location; detecting, basedon analyzing the sensor data, a characteristic of the customer;comparing the characteristic of the customer to a data store storingcharacteristics for a plurality of customers; and identifying a customerprofile based on the comparison, wherein the customer profile matchesthe characteristic of the customer.
 3. The method of claim 2, wherein acharacteristic of the customer is a visual characteristic correspondingto facial features of the customer.
 4. The method of claim 1, furthercomprising: detecting, based on analyzing sensor data corresponding tothe customer interaction location, a set of characteristics of thecustomer; comparing the set of characteristic of the customer to storedcharacteristics for a plurality of customers; and identifying a customerprofile based on multi-factor matching of the set of characteristics tothe stored characteristics.
 5. The method of claim 4, wherein the set ofcharacteristics of the customer includes one or more of: biometric datacollected from the customer via biometric sensors; a height measurementof the customer collected by a camera; a gait profile of the customercollected by the camera; ear features of the customer collected by thecamera; or a voiceprint of the customer collected by a microphone,wherein a customer profile is determined from customer characteristicsstored in a data store based on the biometric data, the heightmeasurement, the gait profile, the ear features, or the voiceprint. 6.The method of claim 1, wherein projecting an augmented reality elementcomprises: projecting the customer-specific interaction data generatedfor the customer on the first surface of the transparent panel, whereinthe customer-specific interaction data is visible to the customerservice representative located in a first area adjacent to a firstsurface of a transparent panel, and wherein the customer-specificinteraction data is prevented from being viewed by the customer locatedin a second area adjacent to a second surface of a transparent panel. 7.The method of claim 6, further comprising: tracking movements of thecustomer; and projecting the augmented reality element including thecustomer-specific interaction data on a location on the transparentpanel such that the customer-specific interaction data is visible to thecustomer service representative while the customer is visible to thecustomer service representative.
 8. The method of claim 7, furthercomprising: moving the location of the augmented reality elementincluding the customer-specific interaction data projected on thetransparent panel as the customer moves; and maintaining a relationshipbetween the location of the augmented reality element including thecustomer interaction data projected on the transparent panel and aposition of the customer.
 9. The method of claim 6, wherein projectingthe augmented reality element including the customer-specificinteraction data to the customer service representative comprises:determining an eye gaze location of the customer service representativeon a first surface of the transparent panel; and projecting theaugmented reality element including the customer-specific interactiondata at the determined eye gaze location of the customer servicerepresentative on the first surface of the transparent panel.
 10. Themethod of claim 6, wherein projecting the augmented reality elementincluding the customer interaction data to the customer servicerepresentative comprises: determining a location on the transparentpanel at which the augmented reality element including thecustomer-specific interaction data is to be projected, wherein thelocation is based on a first location of the customer servicerepresentative in the area adjacent to the first surface of thetransparent panel, and a second location of the customer in the areaadjacent to the second surface of the transparent panel.
 11. The methodof claim 1, further comprising: retrieving data from a data store havinga plurality of user profiles with user characteristics stored thereinfor a plurality of users determining strategies for interactions withthe customer based on the retrieved customer-specific interaction data,wherein the strategies for interaction include a product or serviceoffering, or identification of a security risk; and transmitting thestrategies for interactions to an augmented reality system of a customerservice representative at the customer interaction location.
 12. Anaugmented reality system, comprising: a transparent panel having a firstsurface and a second surface opposite the first surface; a camera deviceconfigured to capture data of a customer; a projection system configuredto project customer interaction data on the transparent panel; and aprocessor configured to: identify the customer at a customer interactionlocation based at least in part on captured data from the camera device;retrieve customer-specific interaction data; determine locations of acustomer service representative in an area adjacent to a first surfaceof a transparent panel and the customer in an area adjacent to a secondsurface of the transparent panel; and project an augmented realityelement on the first surface of the transparent panel, the augmentedreality element displaying the customer-specific interaction data. 13.The augmented reality system of claim 12, wherein the processor isfurther configured to: receive sensor data corresponding to the customerinteraction location from the camera device, the sensor data includingvisual data; analyze the sensor data to detect a customer within aproximity of the location; detect, based on analyzing the sensor data, acharacteristic of the customer; compare the characteristic of thecustomer to a data store storing characteristics for a plurality ofcustomers; and identify a customer profile based on the comparison,wherein the customer profile matches the characteristic of the customer.14. The augmented reality system of claim 13, wherein a characteristicof the customer is a visual characteristic corresponding to facialfeatures of the customer.
 15. The augmented reality system of claim 12,wherein the processor is further configured to: detect, based onanalyzing sensor data corresponding to the customer interactionlocation, a set of characteristics of the customer; compare the set ofcharacteristic of the customer to stored characteristics for a pluralityof customers; and identify a customer profile based on multi-factormatching of the set of characteristics to the stored characteristics.16. The augmented reality system of claim 15, further comprising:biometric sensors; and a microphone, wherein the set of characteristicsof the customer includes one or more of: biometric data collected fromthe customer via biometric sensors; a height measurement of the customercollected by a camera; a gait profile of the customer collected by thecamera; ear features of the customer collected by the camera; or avoiceprint of the customer collected by a microphone, wherein theprocessor is further configured to determine a customer profile fromcustomer characteristics stored in a data store based on the biometricdata, the height measurement, the gait profile, the ear features, or thevoiceprint.
 17. The augmented reality system of claim 12, wherein theprocessor is further configured to: cause the projection system toproject the augmented reality element including the customer-specificinteraction data generated for the customer on the first surface of thetransparent panel, wherein the customer-specific interaction data isvisible to the customer service representative located in a first areaadjacent to a first surface of a transparent panel, and wherein thecustomer-specific interaction data is prevented from being viewed by thecustomer located in a second area adjacent to a second surface of atransparent panel.
 18. The augmented reality system of claim 17, whereinthe camera device is further configured to track movements of thecustomer, and wherein the processor is further configured to cause theprojection system to project the augmented reality element including thecustomer-specific interaction data on a location on the transparentpanel such that the customer-specific interaction data is visible to thecustomer service representative while the customer is visible to thecustomer service representative.
 19. The augmented reality system ofclaim 18, wherein the processor is further configured to cause theprojection system to: move the location of the augmented reality elementincluding the customer-specific interaction data projected on the firstsurface of the transparent panel as the customer moves; and maintain arelationship between the location of the augmented reality elementincluding the customer interaction data projected on the first surfaceof the transparent panel and a position of the customer.
 20. Theaugmented reality system of claim 17, wherein the processor is furtherconfigured to: determine an eye gaze location of the customer servicerepresentative on the first surface of the transparent panel; andproject the augmented reality element including the customer-specificinteraction data at the determined eye gaze location of the customerservice representative on the first surface of the transparent panel.21. The augmented reality system of claim 17, wherein the processor isfurther configured to: determine a location on the transparent panel atwhich the augmented reality element including the customer-specificinteraction data is to be projected, wherein the location is based on afirst location of the customer service representative in the areaadjacent to the first surface of the transparent panel, and a secondlocation of the customer in the area adjacent to the second surface ofthe transparent panel.
 22. The augmented reality system of claim 12,wherein the processor is further configured to: retrieve data from adata store having a plurality of user profiles with user characteristicsstored therein for a plurality of users determine strategies forinteractions with the customer based on the retrieved customer-specificinteraction data, wherein the strategies for interaction include aproduct or service offering, or identification of a security risk; andtransmit the strategies for interactions to the augmented reality systemof a customer service representative at the customer interactionlocation.